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SUMMARY 

In  this  paper  we  Indicate  how  a  class  of  control  processes 
requiring  rr.ultl— dimensional  sequences  of  functions  when 
treated  by  the  direct  methods  of  dynamic  programnlng  can,  by 
means  of  a  transformation  familiar  to  the  theory  of  linear 
functional  equations  of  differential,  difference,  or 
differential— difference  type,  be  reduced  to  problems  involving 
sequences  of  functions  of  one  variable  In  a  number  of  cases, 
and  sequences  of  functions  of  two  variables  In  others. 

lYiese  results  open  the  door  to  a  systematic  study  of  non¬ 
linear  control  processes,  with  or  without  time-lags  and  other 
types  of  hereditary  behavior,  by  way  of  the  method  of 
successive  approximations. 

As  an  example,  we  consider  the  problem  of  minimizing 

J(v)  -  |u(lO) I  +  X  /  v^dt 

'0 

over  all  functions  v(t)  satisfying  the  constraint 
—  .1  <  v(t)  <0,  0  <  t  <  10,  where  u"  .3u'  .02u  •  0, 

u(0)  -  2,  u'(0)  -  -.:5. 


P-1155 

&-15-57 

-11- 

CONTENTS 

SUMMARY .  1 

Section 

1.  INTRODUCTION .  1 

2.  A  GENERAL  FEEDBACK  CONTROL  PROBLEM .  2 

3.  DYNAMIC  PROGRAMMING  APPROACH .  4 

4.  LAGRANGE  MULTIPLIER .  6 

5.  DISCUSSION .  7 

6.  PRELIMINARIES  ON  LINEAR  DIFFERENTIAL  EQUATIONS.  .  8 

7.  A  TEPJIINAL  CONTROL  PROBLEM .  9 

8.  A  GENERAL  CONTROL  PROBLEM  WITH  LINEAR  FEATURES.  .  12 

9.  A  NUMERICAL  EXAMPLE . 15 

10.  MULTI-DIMENSIONAL  VERSION . 17 

11.  USE  OP  SUCCESSIVE  APPROXIMATIONS . l8 

12.  TIME-iAG  PROBLEMS . 21 

BIBLIOGRAPHY . 22 


P-1155 

8-1^7 

-1- 


DYNAMIC  PROORAMMINO  AND  THE  COMPUTATIONAL 
SOLUTION  OP  FEEDBACK  DESIGN  CONTROL  PROBLEMS 

Richard  Bellman 


1.  INTRODUCTION 

In  this  paper  we  wish  to  provide  an  expository  account  of 
some  new  dynamic  programming  techniques  we  have  recently 
developed  for  the  express  purpose  of  handling  variational 
problems  arising  In  the  design  of  feedback  control  circuits. 

Assuming  that  the  reader  Is  familiar  with  the  physical 
background  of  these  questions,  we  sh.all  concern  ourselves 
exclusively  with  some  of  the  mathematical  problems  Involved. 
Detailed  accounts  of  these  processes  may  le  found  In  the 
recent  books  by  Lanlng  and  Battln,  [lb']  »  Truxal,  [l7]  . 

Interesting  accounts  of  other  amalytlc  techniques  used  to 
study  problems  In  the  theory  of  control  may  be  found  In  Bass, 
[ij  ,  Booton,  [l^]  ,  and  Kalman,  [15]  • 

CXir  aim  Is  to  provide  a  straightforward  method  for  ob¬ 
taining  the  numerical  solution  of  various  classes  of  control 
problems,  based  upon  the  use  of  a  modem  digital  computer. 
Although  a  certain  amount  of  analytic  preparation  Is  required, 
necessitating  the  fundamentals  of  the  theory  of  linear 
differential  or  difference  equations,  our  fundamental  purpose 
Is  to  furnish  a  computational  technique  wh.lch  ypasses  any 
detailed  analytic  study  of  the  equations  governing  the  process 
under  consideration. 

In  order  to  keep  the  paper  as  self-contained  as  possible. 
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we  shall  present  a  brief  exposition  of  the  functional  equation 
technique  of  dynamic  programming.  A  more  detailed  account  of 
this  theory  may  be  found  in  our  book,  [2]  ,  and  the  accounts  in 
Qj]  and  [43,  may  also  be  helpful.  Further  mathematical 
discussion  is  contained  in  [5]  ,  H  »  W  • 

We  have  included  a  simple  example  of  the  application  of 
these  techniques,  in  order  to  illustrate  the  time  and  effort 
required  in  handling  these  problems. 

2.  A  GENERAL  FEEDBACK  COHTROL  PROBLEM 

Let  us  begin  by  fonmilating  a  general  control  problem, 
restricting  ourselves,  however,  for  the  sake  of  notational 
simplicity,  to  systems  governed  by  second  order  differential 
equations. 

Given  an  equation  of  the  form 

(1)  ^  -  g(3^,u)  +  v(t),  u(0)  -  Cj,  u’(0).C2, 

dt 

where  v(t)  is  a  forcing  function  or  control  function, 
resulting  from  feedback,  we  wish  to  detennlne  v(t)  In  such  a 
way  as  to  minimize  an  expression  of  the  form 

(2)  J(v)  -  h(u,^)dt  q(v)dt  +  P(x(T),)t'(T)). 

TTie  function  v  will  usually  be  subject  to  additional  con¬ 
straints  of  the  form 
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a.  m<v(t)<m',  0<t<T, 

(3) 

b  .  r(  v)dt  <  b . 

Wi\  — 


In  a  number  of  situations,  these  terms  have  the  following 

Interpretation.  The  first  Integral,  /  h (u,du/dt )dt , 

% 

represents  the  cost  Incurred  due  to  the  deviation  of  the 
system  from  some  desired  operating  state  over  the  time  period 

-  m 

[0,T] ;  the  second  Integral,  j  q(v)dt,  represents  the  cost 
of  using  control  over  that  time  period;  tht  third  term  repre¬ 
sents  an  additional  cost  due  tc  the  final  state  of  the  system 
at  the  end  of  the  process. 

If  the  first  term  Is  absent,  we  speak  of  a  terminal 
control  process.  Actually,  there  la  no  cleai>-cut  distinction, 
since  Various  transformations  will  convert  one  type  of  process 
Into  another. 

The  restrictions  on  v  are  of  physical  nature,  corres¬ 
ponding  to  constraints  concerning  the  maximum  voltage  or 
current  that  can  be  used,  or  in  physical  control  processes,  to 
a  raaxlnmm  angle,  maximum  rate  of  turn,  and  so  on.  Finally, 
the  last  constraint  corresponds  to  an  energy  constraint  on  the 
system,  or  In  economic  terms,  a  constraint  on  the  quantity  of 
resources  available  for  control. 

The  discussion  In  [^J  shows  that  there  Is  a  complete 
analogy  between  feedback  control  problems  for  engineering  and 
economic  control  processes,  and  the  same  mathematical  tech)— 
nlques  can  be  used  In  each  domain. 

b 
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Problems  of  the  foregoing  type  are.  In  general,  extremely 
difficult  to  solve  analytically  In  traditional  fashion,  l.e.  In 
terms  of  explicit  solutions  of  differential  equations,  due  not 
only  to  the  nonlinearity  of  the  equations  but  also,  even  In 
linear  cases,  to  the  presence  of  constraints.  A  discussion  of 
various  classes  of  problems  of  this  type,  together  with  some 
explicit  solutions,  may  be  found  In  Ql2]  and  [l3]  . 


3.  DYNAMIC  PROORAMWINQ  APPROACH 

Let  us  now  briefly  sketch  the  application  of  the  theory 
of  dynamic  progranmlng  to  the  computational  solution  of  prob¬ 
lems  of  the  above  type.  Introduce  the  fun iJ: Ion  of  tliree 
variables  f(c^,C2,T)  by  means  of  the  relation 

(1)  f(cj,c  ,T)  -  Kin  J(u), 


where  J(u)  Is  as  In  (2.2)  and,  for  the  moment  the  function 
satisfies  only  the  constraint  (2.3a). 

The  quantities  c^  and  c^  are  the  state  variables 
specifying  the  Initial  condition  of  the  system  and  T  Is  the 
duration  of  the  process. 

Under  various  conditions,  which  are  usually  met  In  appli¬ 
cations,  we  obtain  for  f  the  nonlinear  partial  differential 
equation 


(2) 


<yf 

3T 


-  Min 


^  ^2  6(C;^,C2)] 


<9f 


where  the  quantity  v  ranges  over  the  Interval  [m,m*]  .  The 
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Initial  condition  Is 
(3)  f(c^,c^,0)  s  0. 

For  the  details  of  this,  see  [2] . 

This  equation  can  be  treated  computationally  by  means  of 
the  usual  approximation  to  the  partial  derivatives  by  partial 
differences.  It  has  turned  out  in  practice,  however,  that  It 
Is  better  to  proceed  as  follows.  In  place  of  the  continuous 
process  described  by  the  equation  In  (2.1),  which  we  write  In 
the  form 

^  -  w,  u(0)  -  c^, 

(M 

^  -  g(u,w)  d-  V,  w(0)  -  c^, 

(a  phase  space  representation),  we  consider  the  discrete 
process  described  by  the  difference  equations 

\*1  -  \  ‘  “k^'  “o  -  =1> 

(5) 

"k+l  -  “k  “  tsK-^k’  ''kJ^'  “O  “  "2' 

k  -  0,  1,  2,  . . . ,  N-1  . 

Here 

(6)  u^  -  u(k6),  -  v(k5),  w^  -  w(k6),  k  -  0, 1 , 2 , . . . ,N-1 , 

and  N6  ■  T. 

In  place  of  minimizing  the  functional  J(u),  we  minimize 
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the  function  of  N  variables 

(7)  , . . . , )  " 


N-1 

Jo 


“k' 


+  6 


N-1 

Jo 


q(v^) 


The  principle  of  optimality,  cf .  [2j  ,  yields  In  place  of  (2) 
the  relation 

fj^(ci,C2)  -  Min  ^6  h(cj,C2)  +  6 


(8) 

where  the  sequence 

( 9 )  ^  » ^2  ^ 


fj^_^  ( c Ls ( c c 2  ^ 

{^K^^l'^2^}  defined  by 
■  Min  K— 1  ^  *  K  ^  1 . 


4.  LAORANQE  MULTIPLIER 

In  order  to  take  account  of  the  constraint  of  (2.5b) 
without  adding  another  state  variable,  we  Introduce  a  Lagrange 
multiplier,  cf .  [8  ]  ,  and  consider  the  new  function 


N-1 

J(vo#Vi, .  . .  ,Vjj-i)  -  6  h(u^,Wj^) 

(1) 

N-1 

^  ^  I  [q(vj^)  -  >^r(v  )] 

k-0  ^  ^ 


For  each  fixed  X,  we  have  a  problem  which  may  be  considered 
as  above.  The  quantity  X  Is  varied  until  the  solution 
corresponding  to  the  constraint  (2.5b)  Is  obtained. 
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5.  DISCUSSION 

The  methods  obtained  above  permit  us  to  obtain  a  compu¬ 
tational  solution  of  the  variational  problem  posed  in  h2. 

The  first  member  of  the  sequence, 
determined  by  the  relation 

(1)  f^(c^,C2)  -  Min  [6  h(c^,C2)  &  g(vQ)]  , 

''o 


and  the  succeeding  members  by  means  of  (5.8).  The  computation 
of  the  maximum  and  the  tabulation  of  the  values  Is  to  be  done 
by  a  digital  computer. 

As  we  might  Imagine,  there  la  a  bit  more  to  the  problem 
than  what  Is  apparently  contained  above.  Leaving  aside  the 
Important  questions  of  grid  size  and  stability,  let  us  examine 
the  memory  problem.  At  each  stage  of  the  computation,  the 
digital  computer  Is  engaged  In  computing  the  values  of  the 
function  fj^(c^,C2),  based  upon  the  values  of 
Consequently,  It  must  store  both  sets  of  values  In  Its 
memory,  together  with  v^  ■  v(c^,C2). 

Consider  the  case  In  which  we  take  a  range  of  ICX) 
admissible  values  for  c^  and  a  like  number  for  Cp.  TVie 
range  of  pairs  (c^,Cp)  will  then  encompass  10,(X)0  values. 

This  Is  a  large  number  even  by  modem  standards.  Nevertheless, 
we  have  resolved  problems  of  this  type  by  these  methods. 


If,  however,  we  consider  problems  Involving  two  coupled 
systems,  we  meet  four-dlmenslonal  systems,  l.e.  systems  In¬ 
volving  four  state  variables.  ITie  approach  outlined  above 
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g 

would  lead  to  a  grid  of  10  values. 

Although  there  are  a  variety  of  techniques  which  can  be 
used  to  reduce  these  formidable  numbers.  It  Is  clear  that  If 
we  wish  to  tackle  really  large  scale  systems,  we  must  Intro¬ 
duce  some  basic  modifications  of  the  method  described  above. 

In  the  remainder  of  the  paper  we  shall  discuss  some  basic 
innovations . 

6.  PREXIMIKARIBS  ON  LINEAR  DIFFERENTIAL  EQUATIONS 

In  this  section  we  collect  some  results  concerning  the 
solution  of  linear  inhomogeneous  equations  which  we  shall  re¬ 
quire  for  our  further  discussion.  Proofs  of  these  results  may 
be  found  In  [9]  . 

Let  us  consider  first  the  case  of  second  order  equations, 
beginning  with  those  with  constsmt  coefficients.  If  u  Is 
the  solution  of  the  equation 

(l)  u"  a^u'  a^u  -  z(t),  u(0)  -  c^,  u'(0)  - 


It  may  be  written  In  the  form 


where  r^  and  r^  are  the  roots  of  the  characteristic 
equation 

r  +  a^r  ■ 


(3) 


0, 
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assumed  distinct  for  our  purposes.  TYie  function  Uq  Is  the 
solution  of  the  homogeneous  equation,  having  the  form 

(4)  u  -  c^u^  +  c^u^, 

where  u^  and  u^  are  the  particular  solutions  of  the  homo¬ 
geneous  equation  detenrilned  by  the  Initial  conditions 

u^(0)  -  1,  u^CO)  -  0, 

(5) 

u^'(O)  .  0,  u^'iO)  -  1. 

If  a^^  and  a^  are  functions  of  t,  u  possesses  a 
representation  of  the  form 

(6)  u  ■  u^  k(t,s)z(8)d8 

u  Cq 

where 

u, ( t )u^(3 )  -  u  { t )u  (s  ) 

(7)  k(8)  .  ? - 2 - 1 -  . 

U^  U^(8 )Up'  (s) 

The  solutions  u^  and  u^  are  determined  by  the  boundary 
conditions  of  ( 5 )  • 

Corresponding  results  hold  for  the  solutions  of  Inhomo¬ 
geneous  linear  differential  equations  of  all  order,  and  are 
most  easily  obtained  via  vectorwnatrlx  operations,  cf.  [9]  » 
Chapter  2. 

7.  A  TERMINAL  CONTROL  PROBLEM 


Let  us  now  consider  the  following  problem: 


"Minimize 
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(1)  |u(T)|  +  X/’’'  v^dt 

over  all  v(t)  satisfying  m  <  v(t)  <  m',  0  <  t  <  T,  where 

(2)  u"  +  au'  -f  bu  -  v(t),  u(0)  •  u’(0)  ■  ^2*" 

As  we  know  from  the  previous  sections,  this  problem  can  be 
treated  computationally  by  means  of  sequences  of  functions  of 
two  variables.  Let  us  now  show  that  It  can  be  reduced  to  a 
computational  problem  Involving  sequences  of  functions  of  one 
variable . 

As  in  ^6,  the  solution  of  (2)  can  be  written 
(5)  ^  k(t-e  )v(3)ds, 

where  Uq  «  c,u^^^  +  c^u^^^,  the  solution  of  the  homogeneous 

rit  Tpt 

equation,  and  k(t)  is  a  linear  combination,  b^e  -f  b^e 
a  known  function. 

The  problem  posed  above  can  then  be  written: 

"Minimize 

(^)  J(v)  -  I  z  +  k(T— 6  )v(  s  )ds  I  +  X  v^dt, 

0  ^0 

over  all  functions  v(t)  satisfying  m  <  v(t)  <  m’." 

Let  us  Introduce  the  function  of  z  and  T  defined  by 


the  relation 
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(5)  f(z,T)  -  Min  J(v), 


Khere  m<  v(t)  <m',  0<  t  <T. 

To  obtain  a  functional  equation  for  f(2,T),  we  argue 
ae  follows.  Suppose  that  v(8)  has  been  chosen  In  an  Initial 
Interval  [^,5] •  Then 


J(v)  •  |z  +  k(T— s)v(s)d3  +  /  k(  T— «  )  v(  8  )ds  1 

^  0  5 


'y  f)  O  -V  ^  ^ 

>  X  /  V  dt  +  X  /  '  v'^dt 

^  (  c. 

0  0 

(6) 

-  ,2  +  k(T— e)v(s)ds  ■♦■  /'  k(  T— 5— s  )  v(s"*-6  )d8  | 

^0  ^0 


X  v^dt  +  X  v^(s'f5)d8. 

-'0  "0 


It  Is  clear  that  no, matter  how  v(s)  has  been  chosen  in  [0,b]  , 
we  will  choose  v(s)  over  so  ae  to  mlnlr.lze  the  second 

expression  In  (r).  Tills  Is  a  particular  'ase  of  the  principle 
of  optimality,  cf.  [2\  Hence,  for  any  choice  of  v(s)  In 
[0,6_!  ,  an  optimal  continuation  yields 


(7)  J(v)  -  f(z  ^  k(  T— a  )v(  8  )d8  ,T-5  )  -f  X  v^dt. 

‘'0  0 


Consequently,  we  choose  v(s)  in  Il0,5j  so  as  to  minimize 
this  expression,  obtaining  the  functional  equation 


f(z,T)  -  Min  f(z  +  ,  l<(T-s)v(s)  J0  ,T-5  )  >  X  v^dt '  . 

v[0,5]  ^  ^0  ‘O  j 


(6) 


Por  computational  purposes,  we  replace  this  by  the 
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approxlmate  equation 

(9)  f(z,T)  -  Min  [f(2-»'Vk(T)6,T-6)  X6v^  , 

V 

where  m  <  v  <  m'  and  T  •  0,  6,  26,  . . . ,  with  f (z,0)  -  |  z  1  . 

We  have  thus  obtained  the  desired  objective  oV  replacing 
a  numerical  computation  Involving  a  sequence  of  functions  of 
two-dimensional  functions  by  one  Involving  a  sequence  of  one— 
dimensional  functions. 

8.  A  GENERAL  CONTROL  PR0BLg<  WITH  LINEAR  FEATURES 
Let  us  now  consider  the  following  problem: 

"Minimize 

(1)  J(v)  i^(t)u(t)  +  g2(t)u' ( t  )]dt  +  b^u(T)  +  b^u'lT) 

over  all  v(t)  satisfying  the  constraints 

(a)  m<v(t)<m’,  0<t:<T, 

(2) 

(b)  r(v(t))dt  <  k, 

0 

amd  the  relation 

(3)  u"  a(t)u'  +  b(t)u  -  V,  u(0)  -  c^,  u'(0)  -  c^." 

Let  us  first  show  that  this  problem  Is  equivalent  to  that 


of  minimizing  a  linear  functional 
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(^)  J,  (v)  -  h(t  )v(t  )dt 

^  ‘'0 

over  all  v(t)  satisfying  the  constraints  of  (2),  where 
h(t)  Is  a  known  function. 

To  see  this,  we  write  u(t)  In  the  form 

(5)  u(t)  -  c.u  (t)  >  L'  u^(t)  +  k(t,s)v(s)d8, 

11  ^  ^ 


where,  as  we  know  from  t^6, 

u.  ( t)u.^(3)  -  u^(t)u,  (s) 

(6)  k(t,3)  -  J: - ^ ^ i - 

w(8  ) 


'lYien 


(7) 


+  b,  k(T,s)v(3)ds  ‘  k  (  T ,  s  )  v(  s  )  ds  . 

ly'Q  ^  t/Q  t 


v)  -  c,  +  k(t.s)v(s)d8 

^  0  1  0 


+  go(t) 


o(t)  /  ’  k  (t,s)v(s)dB  dt 


The  last  two  Integrals  hiave  the  desired  form.  Let  ua 
show  that  th.e  first  Integral  may  also  be  written  In  this  way. 


We  have 
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(8) 


gl(t)  k(t,8)v(8)d8^  dt 


■•r  U'  ^ 


U,(t)u2(8)  -  U2(t)u^(8)' 


w(8) 

^  U^(s)v(8) 

w(8) 


v( 8 )d8  )  dt 


>  .  .  /)t  U,(3)v(s) 

g,(t)up(t;  r  -  ds 

^  ^  w(8) 


ds 


dt . 


Integrating  by  parts,  this  becomes 


/ 


T  ru2(8)v(a)  ^ 


0  L  w(8) 


/  ^  g^(t)u^(t )dt 


U.(8)v(8) 

-  J  g,(t)uo(t)dt 

w(8)  8  ^  ^ 


ds . 


We  see  then  that  our  problem  reduces  to  maximizing 
as  given  by  (4),  subject  to  the  constraints  of  (2).  TTils  Is  a 
problem  which  In  a  number  of  cases  cam  be  solved  analytically 
In  explicit  form  cf.  [l3j  •  However,  In  order  to  Indicate  the 
methods  we  would  employ  In  hlgher-Kllmenslonal  cases  where 
there  were  several  v^^,  let  us  discuss  the  functional  equation 
approach . 

Consider  the  function  f(k,z)  defined  by 

(9)  f(k,z)  -  Min  h(t)v(t)dt 

V  z 


where  v  satisfies  the  constraints 
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(a)  m<v(t)<m’,  0<t<T, 

(10) 

(b  )  r(  v(  t )  )dt  <  k. 

z 


'IVien,  as  In  ^7,  It  Is  easy  to  obtain  the  functional 
equation 


(11) 


f(k,z)  -  Min  \  h(t)v(t)dt 

m<v^’  z 

+  f(k  r(v(  t)  )dt  ,z+6)  [  . 

2  J 


For  computational  purposes,  t  .Is  reduces  to 

(12)  r(k,2)  «  Min  [h(z)v6  4  f  ( k— r(  v  ) 5  , z-fS  )]  , 

with  f(k,T)  s  0. 

9.  A  NUMERICAL  EXAMPLE 

As  an  example  of  the  foregoing,  consider  ttie  problem  of 
determining  v(t)  so  as  to  minimize 

(1)  'u(io)i  +  v^dt. 


where 

(2)  u"  4  .^u'  4  .02^J^  -  V,  u(0)  -  2,  u'(0)  -  —  .3. 


The  restrictions  upon  v  arc 


(3) 


-  .1  <  v(t)  <  0. 
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Uslng  the  methoda  of  ^7,  we  convert  the  problem  Into  that 
of  computing  the  sequence 


(4) 


f^Q(a)  -  lai , 


f^(a)  -  Min  [Xv^  At  +  f (a+k(  10-t )v^t )] 


The  function  k(s)  la  defined  by 

(5)  k(s)  -  10(e-^°  - 

The  solution  of  the  homogeneous  linear  equation 
u"  +  .3u'  .02u  -  0,  with  the  assigned  boundary  conditions. 

Is  given  by 

(6)  u(t)  -  e”-^^  + 

so  that  u(lO)  "  e”^  +  e~~^  ^  .50. 

We  wish  to  determine  fQ(.50). 

For  computing,  we  choose  the  parameters  At  -  .2  sec, 

Aa  ■  .01,  which  divides  0  £  a  <  50  Into  50  parts,  X  -  1.0, 
and  allow  v(t)  to  range  between  -.1  and  0.  The  values  of 
v(t)  In  this  Interval  are  taken  at  Intervals  of  .01  between 
—  .10  and  —.0)  and  In  Intervals  of  .003  between  —.03  and 
0. 

The  optimal  policy  yields  a  value  of  .OO56  for 
v^dt,  and  u(lO)  -  .0002,  so  that  fQ(50)  -  .OO58.  The 
optimal  choice  of  v(t),  smoothed  In  order  to  compensate  for 
our  restriction  to  discrete  values  of  v  Is  shown  below. 
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Pour-polDt  Interpolation  over  a  leas  dense  a-grld  yields 
the  same  results.  The  graph  of  — v(t)  Is  given  below. 


L.  ■  _ _ I _ _ i, _ I - 1 - - - ^ 

0  2  4  6  e  10 

t 

PIQURE  1 

The  answer  required  two  hours  on  the  RAND  JOHNNIAC  Computer, 
using  a  floating  point  Interpretive  scheme.  The  same  calcula¬ 
tions  would  consume  fifteen  minutes  1''  an  IBK-704  or  comparable 
computer  were  used. 

10.  MUI.TI-DIMENS TONAL  VERSION 

Let  us  make  two  Important  observations. 

I.  Ihe  problem  of  minimizing  a  given  function 

h(x^(T),  x^lT),  x^(T))  over  all  v^(t)  satisfying 

the  constraints 

( a  )  ( t )  <  ,  1  -  1 ,  2 ,  .  .  .  ,  N ,  0  <  t  <  T 

(1) 

T 

(t )  /'  ?(v, . . . ,v  )dt  <  b, 

"0  n  — 
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and  the  relations 
dx.  N 

^  ’  A  *  ''i’  ^  ■■■’ 

x^(0)  -  c^,  can  always  be  treated  by  meemB  of  sequences  of 
functions  of  k  variables. 

If  N  Is  large  and  k  ■  1  or  2,  this  means  that  we  have 

a  feasible  method  of  solution  using  this  new  approach,  where 

we  could  not  treat  the  problem  previously. 

II.  If  the  problem  Is  that  of  minimizing 

"  N  "  N 

(3)  J(v)  -  X  (t)  dt  +  I 

0  L  J  A  ^  ^ 


the  problem  may  be  treated  by  means  of  sequences  of  functions 
of  one  variable,  regardless  of  the  size  of  N. 


11.  USE  OP  SUCCESSIVE  APPROXIMATIONS 


We  have  seen  In  the  previous  sections  that  the  problem  of 
minimizing  a  functional  of  the  form 


J(v)  -  / 
0 


^  bi(t)x^(t)  dt  + 


over  all  functions  v^(t)  satisfying  the  constraints 

(a)  m^<v^(t)<m^',  1-1,  2,  ...,N,  O^t^T, 

(2) 

/T 

^  g(v2^,V2,...,Vj^)dt  ^  x. 


where  the  are  determined  by  linear  homogeneous  equations 

of  the  form 
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(5) 


dx.  N 
•J  *  1 

Xi(0)  -  c^, 


can  be  resolved  numerically  by  means  of  sequences  of  functions 
of  one  variable,  regardless  of  the  magnitude  of  N. 

Essential  In  this  reduction  was  the  superposition 
property  of  linear  systems.  Let  us  now  Indicate  how  the 
method  of  successive  approximations  may  be  applied  to  yield  an 
approach  to  nonlinear  systems  by  meajis  of  thes?  methods. 

Consider  the  problem  of  minimizing  a  functional  of  the 

form 

(^)  J(v)  P(x^,X2, . .  .  ,Xjj)dt  0(x^(T),X2(T),  ...,Xj^(T 

over  all  v^^  satisfying  (2),  where 
dx. 

W  •  . 1  -  1-  ^ . "> 

(5) 

Xi(0)  -  c^. 

Let  v^^®^(t),  1-1,  2,  ...,N,  be  an 

Initial  choice  of  the  v^(t),  and  x^^^^(t),  1-1,  2,  ...,  N, 
the  values  of  x^  determined  by  (5)*  Consider  the  new  problem 
of  minimizing  the  linear  functional 
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(6) 


L 


Fix  X  X 

^  V  ,^2  » • • • »^N  ' 


^  ^  (  0  V  ^P 


“I 


Jx, 


nrr 


dt 


♦  0{xj<°'(T).Xj*°^'r),...,x^*°>(T)) 


N 


+  2  (Xj(T)-Xj<°'(T))  ^ 


l/X, 


m 


subject  to  the  linear  equations 


(7) 


.  H  (x  <0>  X  <°>  X  <°)l 


^  !  (®)\  '^H  .  ( ^  \ 


x^(0)  -  c^,  1  -  1,  2,  ...,  N.  The  functions  v^(t),  as 

before,  satisfy  (2). 

As  a  linear  problem,  the  minimizing  v^(t)  can  be 
determined  by  means  of  sequences  of  functions  of  one  variable 
The  numerical  solution  of  the  foregoing  problem  Is  thus  a 
relatively  simple  matter. 

Having  detentilned  the  new  functions  and  x, 


In  this  fashion,  we  replace  and  x,(°) 

and  continue  the  process. 


by  V, 


^1 

(1) 


The  original  nonlinear  variational  problem  has  thus  been 


replaced  by  a  sequence  of  linear  variational  problems.  Con¬ 
sequently,  the  original  nonlinear  variational  problem. 
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requlrlng  a  sequence  of  functions  of  N  variables  for  Its 
solution,  has  been  replaced  ly  a  sequence  of  problems  requiring 
functions  of  one  variable. 

TVie  question  of  the  convergence  of  this  sequence  of 
solutions  to  the  solution  of  the  original  problem  Is  one  wtilch 
requires  a  detailed  ti'eatment  wh.lch  we  shall  not  present  here. 

12.  TIME-LAQ  PPORLSMS 

If  the  underlying  equation  of  th.e  system  has  the  fonn 

(1)  -  g(^,  u(t),  u(t-t^),  u(t-t^)), 

dt 

corresponding  to  time-lags  and  retarded  control,  the  direct 
method  of  dynamic  programming  presented  In  5  yields  no 
feasible  computational  scheme. 

If  the  equation  Is  linear,  and  the  criterion  function  Is 
linear,  representation  theorems  akin  to  those  given  In  ^6, 
cf.  [lO]  ,  »  permit  us  to  obtain  results  corresponding  to 

those  obtained  above. 

These  and  further  results  pertaining  to  control  problems 
for  more  general  types  of  functional  equations  will  be 
presented  In  detail  In  other  publications. 
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Roger,  White  ...  understand. 

(TURNS  TO  SBIIOR  DIRECTOR) 

White  has  a  B- 25-type  Bogle  that's  track  Charlie 
29,  no  marking. 

SENIOR  DIR:  •  No  marking?  One  aircraft? 

DIRECTOR  1:  One  aircraft. 

MI  OFFICER:  Charlie  29? 

DIRECTOR  1:  Charlie  29. 

SENIOR  DIR:  Nothing  on  29?  (TO  MI  OFFICER) 

CUT  TO:  (CPB  SHOT) 

NARRATOR:  Good  decisions  are  not  necessarily  all  fast. 

Because  Charlie  29  Is  farther  away,  the  Senior 
Director  tries  to  get  more  Information  before 
opening  fire. 

DIRECTOR  1:  (TO  MI  OFFICER) 

30  has  two  ...  Go  ahead.  Red!  Roger. 

CUT  BACK  TO:  (FULL  STATION  SHOT) 

SENIOR  DIR:  (TURNS  TO  MI  OFFICER) 

What  did  you  get  on  29? 

MI  OFFICER:  We  have  nothing,  sir. 

SENIOR  DIR:  POUNCE  29. 

DIRECTOR  1:  (IN  BACKGROUND) 

Roger,  Red.  Maintain  surveillance. 

CONTROL  TECH:  Kingpin,  we  ordered  a  splash  on  29. 

DIRECTOR  1:  Redblrd  White,  POUNCE.  Over. 

CUT  TO:  (REAR  VIEW  OF  TOP  DECK  WITH  FOUR  EXPERIMENTERS 

IN  FOREGROUND) 


•*  f 


-r 

4’ 
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